Skip to content

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 40 commits into from
Closed

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

wants to merge 40 commits into from

Conversation

TommyBrosman
Copy link
Contributor

@TommyBrosman TommyBrosman commented May 12, 2025 •
edited
Loading

Adds a new Persisted Schema Format with persisted metadata support. This change does not enable it by default.

Changes

  • Add schema FormatV2
  • Register FormatV2 with schema codec
  • Update codec.spec to cover FormatV2 as well as FormatV1
  • Add schema v2 snapshots
  • Persisted metadata object in schema
  • FormatV2-specific tests
  • Documentation

Notes for reviewers

Todo

  • Changeset description
  • simple-tree changes to expose persisted metadata
  • Tests for simple-tree layer
  • Expose new APIs via alpha types
  • Write persisted node and field metadata in the codec
  • Clone and freeze the metadata trait passed to simple-tree

Related PRs

@TommyBrosman
Initial change.
bd8406a 
- Added in-memory schema format v2.
- Implemented the schema format v2 codec.
- Updated test utils that previously assumed there was only one format.
- Added snapshots for many of the v2 tests.
@github-actions github-actions bot added area: dds Issues related to distributed data structures area: dds: tree area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct public api change Changes to a public API base: main PRs targeted against main branch and removed area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct area: dds Issues related to distributed data structures public api change Changes to a public API area: dds: tree labels May 12, 2025
@TommyBrosman TommyBrosman mentioned this pull request May 12, 2025
Closed
@github-actions github-actions bot added area: dds Issues related to distributed data structures area: dds: tree area: examples Changes that focus on our examples area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct public api change Changes to a public API labels May 13, 2025
TommyBrosman
TommyBrosman commented May 13, 2025
View reviewed changes
TommyBrosman
TommyBrosman commented May 13, 2025
View reviewed changes
@github-actions github-actions bot removed the area: examples Changes that focus on our examples label May 13, 2025
@TommyBrosman
- Changed TreeNodeStoredSchema implementations to take the write vers…
b819742 
…ion as a parameter.

- Fixed naming of schema index formats.
- Changed schemaChangeFormat to be schema format-agnostic.
@TommyBrosman
- Changed the stored schema implementation to handle multiple schema …
0b5c842 
…formats.

- Moved SchemaCodecVersion.
@TommyBrosman TommyBrosman marked this pull request as ready for review May 13, 2025 19:49
@TommyBrosman TommyBrosman requested a review from a team as a code owner May 13, 2025 19:49
CraigMacomber
CraigMacomber reviewed May 13, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 13, 2025
View reviewed changes
@github-actions github-actions bot added the public api change Changes to a public API label May 22, 2025
alexvy86
alexvy86 reviewed May 22, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 27, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 27, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 27, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 27, 2025
View reviewed changes
CraigMacomber
CraigMacomber reviewed May 27, 2025
View reviewed changes
TommyBrosman and others added 3 commits May 27, 2025 10:56
@TommyBrosman
Refactor: persistedMetadata -> metadata on persisted types.
6ca189a
@TommyBrosman @CraigMacomber
Apply suggestions from code review
85240dc 
Co-authored-by: Craig Macomber (Microsoft) <42876482+CraigMacomber@users.noreply.github.com>
@TommyBrosman
- Updated the changeset description
8883d45 
- Moved JsonCompatibleReadOnly and JsonCompatibleReadOnlyObject to alpha API status so they can be used in schemas
- Implemented field factory methods and added test coverage
- Added requiredRecursive to schemaFactoryAlpha (not sure if it should be part of this change though)
@github-actions github-actions bot added the area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct label May 28, 2025
TommyBrosman
TommyBrosman commented May 28, 2025
View reviewed changes
@TommyBrosman
Reverted unnecessary change.
9c2c13f
@TommyBrosman
Wired up node schema metadata persistence. Currently errors on tests …
c62f7ec 
…involving FormatV2.
@TommyBrosman
Reverted unneeded change.
2f6b519
@TommyBrosman
Removed tests for the alpha API I removed in a previous revision.
ae1779b
@TommyBrosman
This revision adds simple-tree persistence for node and field schema …
7eff8ab 
…metadata.

- Fixed a bug where FormatV2 was including FormatV1 fields.
- persistedMetadata is now only written when non-undefined.
- simple-tree implementations for individual schema kinds (arrayNode, mapNode, objectNode) now pass persistedMetadata down from the outer factory calls.
- Introduced alpha APIs for simple-tree in places where we need to pass persistedMetadata but don't want to pollute the public API.
- Updated tests to pass persistedMetadata when constructing SimpleNodeSchemas.
- Wrote simple tests for node and field persisted metadata.
TommyBrosman
TommyBrosman commented May 30, 2025
View reviewed changes
packages/dds/tree/src/test/simple-tree/api/schemaFactory.spec.ts
const factory = new SchemaFactoryAlpha("");

const fooMetadata = {
persistedMetadata: { "a": 2 },
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't need to be persistedMetadata here.

TommyBrosman
TommyBrosman commented May 30, 2025
View reviewed changes
packages/dds/tree/src/test/simple-tree/api/schemaFactory.spec.ts
baz: schemaFactory.optionalAlpha(schemaFactory.string, {
persistedMetadata: fooMetadata,
}),
qux: schemaFactory.optionalRecursiveAlpha(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to conform to example usages of the recursive field APIs more closely.

@TommyBrosman
- Fixed missing persistedMetadata on fields. toStoredSchema.ts and sh…
f3f572c 
…aredTree.ts weren't copying them in all cases where the needed to, and I was missing SimpleFieldSchema.persistedMetadata.

- Added the v5 Shared Tree format and associated snapshots.
- Fixed FieldSchemaAlpha.persistedMetadata so it won't return an empty object if the data is undefined.
@TommyBrosman
Merge branch 'main' into metadata-schema-3
faaf2f7
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 2, 2025

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output


> fluid-framework-docs-site@0.0.0 ci:check-links /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test "npm run serve -- --no-open" 3000 check-links

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> fluid-framework-docs-site@0.0.0 serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/

> fluid-framework-docs-site@0.0.0 check-links
> linkcheck http://localhost:3000 --skip-file skipped-urls.txt

Crawling...

Stats:
  222981 links
    1707 destination URLs
    1939 URLs ignored
       0 warnings
       0 errors

@TommyBrosman
Copy link
Contributor Author

Closing. The changes from this PR ended up in:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexvy86 alexvy86 alexvy86 left review comments

@CraigMacomber CraigMacomber CraigMacomber left review comments

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees
No one assigned
Labels
area: dds: tree area: dds Issues related to distributed data structures area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct base: main PRs targeted against main branch changeset-present public api change Changes to a public API
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@TommyBrosman @CraigMacomber @alexvy86